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Amendments to the Claims : 

Following is a complete listing of the claims pending in the application, as 
amended: 

1-14. (Cancelled) 

15. (Currently amended) One or more computer memories collectively 
containing an advertising targeting data structure, comprising a plurality of entries, 
each entry corresponding to a user and containing: 

information identifying a test group to which the user belongs, the 
identified test group having been selected for the user without regard for user profile 
information.V the^ideritified test group indicating which of a plurality jDfjsequences of 
conditions will', be. applied when an advertising request originating with the user is 
received; and 

for each of the conditions of the indicated sequence of conditions, 
information identifying a treatment subgroup to which the user belongs, the identified 
treatment subgroup having been selected for the user without regard for user profile 
information, the identified treatment subgroup indicating which of a plurality of 
advertising treatments will be applied when the condition is the first condition in the 
sequence of conditions to be satisfied. 

16. (Cancelled) 

17. (Cancelled) 

1 8. (Withdrawn) A method in a computer system for analyzing user targeting 
results, comprising: 

for an advertising targeting program having a plurality of independent 
dimensions, selecting a dimension in which to perform a comparison; 
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for user targeting effectiveness metrics each having a value in each of the 
independent dimensions, aggregating the metrics for each value of the selected 
dimension; and 

comparing the different values of the selected dimension by comparing 
the corresponding aggregated metrics. 

19. (Withdrawn) The method of claim 18 wherein the selected dimension is 
comprised of two or more testing groups each corresponding to a different user data 
analysis approach, and wherein the effectiveness of the different user data analysis 
approaches is compared. 

20. (Withdrawn) The method of claim 18 wherein the selected dimension is 
comprised of two or more advertising messages all displayed to users in the same 
segment, and wherein the effectiveness of the different advertising messages for users 
in the segment is compared. 

21. (Withdrawn) One or more computer memories collectively containing an 
advertising targeting result data structure reflecting the result of targeting using test 
groups, conditions for each test groups, and treatment subgroups for each condition, 
the data structure comprising: 

for each distinct combination of (a) one of the test groups and (b) one 
treatment subgroup for each of the conditions for the test group, an advertising 
effectiveness metric aggregated across all users that are assigned to the test group 
and the treatment subgroup for each of the conditions for the test group, 
such that, to determine a level of effectiveness of the conditions of a test group, the 
effectiveness metrics for users assigned to that test group may be aggregated, 
and such that, to determine a level of effectiveness of a treatment of a treatment 
subgroup, the effectiveness metrics for users assigned to that treatment may be 
aggregated. 
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22. (Withdrawn) The computer memories of claim 21 wherein the 
effectiveness metric is click-throughs. 

23. (Withdrawn) The computer memories of claim 21 wherein the 
effectiveness metric is conversion rate. 

24. (Withdrawn) The computer memories of claim 21 wherein the 
effectiveness metric is average sales. 

25. (Withdrawn) The computer memories of claim 21 wherein the 
effectiveness metric is session length. 



26. (Withdrawn) The computer memories of claim 21 wherein the 
effectiveness metric is user registration rate. 

27-29. (Cancelled) 



30. (New) A method in a computing system for evaluating alternative 
advertising strategies, comprising: 



different sequences of conditions to apply to the^user when an advertising request is 
received for the user, and 



condition sequence selected for the user, randomly selecting one of a plurality of 
different advertising messages associated with the condition to present to the user 
when the condition is the first condition of the sequence that is satisfied by profile 
information for the user; and 




for each of a plurality of users: 

in the computing system, randomly selecting one of a plurality of 





in the computing system, for at least one of the conditions in the 



each time an advertising request is received for one of the plurality of 



users: 
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in the computing system, applying to the user the condition 
sequence that was randomly selected for the user to identify the first condition of the 
sequence that is satisfied by profile information for the user, and 

in the computing system, presenting the user the advertising 
message associated with the identified condition that was randomly selected for the 
user. 

31. (New) The method of claim 30, further comprising: 

in the computing system, generating an assessment of the effectiveness 
of the advertising messages presented to each user; 

in the computing system, for each of the plurality of condition sequences, 
aggregating the effectiveness assessments of the users for whom the condition 
sequence was selected; 

in the computing system, selecting as the most effective of the plurality of 
condition sequences the condition sequence having the highest aggregated 
effectiveness assessment; 

in the computing system, for each of the plurality of advertising message, 
aggregating the effectiveness assessments of the users for whom the advertising 
message was selected; and 

in the computing system, selecting as the most effective of the plurality of 
advertising messages the advertising message having the highest aggregated 
effectiveness assessment. 

32. (New) A method in a computing system for evaluating alternative 
advertising strategies, comprising: 

in the computing system, receiving an advertising request for a user 
during the duration of an advertising campaign, the received request containing an 
identifier uniquely identifying the user; 

if the user has not been added to an advertising test group and 
advertising test subgroup for the campaign: 
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in the computing system, without reference to any profile 
information for the user, selecting an advertising test group among a plurality of 
advertising test groups, each advertising test group of the plurality corresponding to a 
different sequence of conditions, 

in the computing system, assigning the user to the selected 
advertising test group using the identifier contained in the received request, 

in the computing system, without reference to any profile 
information for the user, selecting one advertising test subgroup among a plurality of 
advertising test subgroups that are associated with the advertising test group to which 
the user is assigned, each advertising test subgroup of the plurality corresponding to a 
different set of advertising messages to be presented in response to the application of 
the sequence of conditions corresponding to the advertising test group to which the 
user is assigned, and 

in the computing system, assigning the user to the selected 
advertising test subgroup using the identifier contained in the received request; 

in the computing system, applying to profile information for the user the 
sequence of conditions corresponding to the advertising test group to which the user is 
assigned to identify the first condition of the sequence that is satisfied; and 

in the computing system, presenting to the user an advertising message 
associated with the identified condition that is among the set of advertising messages 
corresponding to the advertising test subgroup to which the user is assigned. 

33. (New) The method of claim 32, wherein the acts of claim 32 are 
performed for a plurality of advertising requests received for a plurality of users, the 
method further comprising: 

in the computing system, generating an assessment of the effectiveness 
of the advertising messages presented to each user; 

in the computing system, for each of the plurality of condition sequences, 
aggregating the effectiveness assessments of the users for whom the condition 
sequence was selected; 
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in the computing system, selecting as the most effective of the plurality of 
condition sequences the condition sequence having the highest aggregated 
effectiveness assessment; 

in the computing system, for each of the plurality of advertising message, 
aggregating the effectiveness assessments of the users for whom the advertising 
message was selected; and 

in the computing system, selecting as the most effective of the plurality of 
advertising messages the advertising message having the highest aggregated 
effectiveness assessment. 

34. (New) The method of claim 32 wherein the selecting acts are performed 
by, in the computing system, selecting an advertising test group and an advertising test 
subgroup that' tend to conform the relative sizes of the advertising test groups and 
advertising test subgroups to predetermined relative sizes for the advertising test group 
sand advertising test subgroups. 
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